Efficient clustered server-side data analysis workflows using SWAMP

Technology continues to enable scientists to set new records in data collection and production, intensifying a need for large scale tools to efficiently process and analyze the growing mountain of data. To complement growth in the number of data centers and the volume of data they store, we introduce our Script Workflow Analysis for MultiProcessing (SWAMP) system. Our system provides safe server-side processing capabilities that allow scientists to reuse familiar desktop-based analysis methods represented in shellscripts. Built-in script compilation isolates file accesses and generates workflows, while a cluster-capable execution engine partitions and executes the resulting workflow. Benchmarks illustrate up to 20X performance gains, as well as the importance of I/O considerations which make other computation systems less effective at geoscience data reduction. Communicated by: H.A. Babaie This work was supported by the National Science Foundation under grant IIS-0431203. D. L. Wang (B) SLAC National Accelerator Laboratory, 2575 Sand Hill Road, M/S 97, Menlo Park, CA, USA e-mail: [email protected] C. S. Zender Dept. of Earth System Science, University of California, Irvine, Irvine, CA, USA D. L. Wang · S. F. Jenks Dept. of Elec. Engn. and Comp. Sci., University of California, Irvine, Irvine, CA, USA